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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address « 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )D Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)IEI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-41 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) IHI Claim(s) 1^41_ is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

1 0)D The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
1 1 )□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected -to by the Examiner. 
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13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
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1 Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 



3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
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14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 
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15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
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DETAILED ACTION 

1. Claims 1-41 are pending in this application. 



Response to Arguments 

2. Applicant's arguments with respect to claims 1 , 12, 16, 20, 24, 27 and 38-39 
have been considered but are moot in view of the new ground(s) of rejection. 
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Claim Rejections - 35 USC § 103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1-3, 11, 27-28 and 31 are rejected under 35 U.S.C. 103(a) as being 

unpatentable over Cullen et al. (US. Patent No. 5,995,978). 

Regarding on claims 1,11 and 27, Cullen teaches a method comprising: 

Initiating a search for images based on a at least one query keyword in an query 

(col. 4, lines 9-13); and 

Cullen does not explicitly identifying, during the search, first images having 

associated keywords that match the query keyword and second images that contain 

low-level features similar to those of the first images. However, Cullen teaches, "search 

proceeds for the text string in the textual portions of the documents contained in the 

database in the text based search step 104. If the text based search yields the 

document of interest, then the user may discontinue any further processing in the step 

106. Otherwise, documents containing the text string become the basic for image 

based search" (col. 4, lines 14-20). This teaches the limitation of the claim. Therefore, 

it would have been obvious to one ordinary skill in the art at the time of the invention 

was made to modify Cullen's system include the keyword and similarity search because 

that would allow Cullen's to retrieve the better results. 
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Regarding on claims 2 and 28, Cullen teaches ranking the first and second 
images (col. 6, lines 30-35) 

Regarding on claims 3 and 31 , Cullen teaches presenting the first and second 
images (fig. 9). 

4. Claims 4-10 and 32-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Cullen et al. (US. Patent No. 5,995,978) in view of Granham et al. 
(US. Patent No. 6,369,811). 

Regarding on claim 4, Cullen teaches presenting the first and second images to 
a user (col. 4, lines 14-27); however, Cullen does not explicitly teach monitoring 
feedback from the user as to which of the first and second images are relevant to the 
query. On the other hand, Graham teaches, "the modification may occur in the 
background or may involve explicit user feedback input. The locations of concept if 
interest determined by pattern identification stage 620 are monitor by profile updating 
stage 634. Profile stage 624 notes the proximity of other keywords and key phrase 
within each analyzed document to the location of concepts of interest, the structure and 
contents of belief system 700 are updated in the back ground without the user input by 
profile updating data" (col. 7, lines 52-65). This teaches monitoring the feedback from 
the user or by the system. Therefore, it would have been obvious to one ordinary skill in 
the art at the time of the invention was made to modify the teaching of Graham into 
Cullen in order to provide relevant feedback from the user based on the user 
assessment of the retrieval images. 
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Regarding on claims 5 and 32, Cullen teaches presenting the first and second 
images to a user (col. 4, lines 14-27); However, Cullen does not explicitly teach 
receiving feedback from the user as to whether the first an second images are relevant 
to the query; and learning how the first and second images are identified based on the 
feedback from the user. On the other hand, Graham teaches, " The modification may 
occur automatically in the back ground or may involve explicit user feedback input" (col. 
7, lines 52-54). This teaches the modification to the keyword to the concept or images. 
In addition, Graham teaches, "profile updating stage 624 notes the proximity of other 
keywords and key phrase within each analyzer document to the locations of concepts of 
interest. If particular keyword and key phrase are always near a concept of interest, the 
structure and content of belief system 700 are updated in the background without user 
input by profile updating stage 624" (col. 7, lines 56-62). This teaches the system learn 
how to assign new keyword to the concept or image. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention was made to employ 
the learning technique of Graham into Cullen because by allowing the system 
intelligently learning from the user feedback to introduce the new connection between 
keyword and image would allow the user efficiently to retrieve better search results. 
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Regarding on claims 6 and 33, Cullen teaches presenting the first and second 
images to a user (col. 4, lines 14-27); however, Cullen does not explicitly teach 
receiving the user feedback for the user as to which of the first and second images are 
relevant to the query; and refining the search to identify additional images that contain 
low-level features similar to those of the images indicated by the user as being relevant 
to the query. On the other hand, Graham teaches, "the modification may occur 
automatically in the background or may involve explicit user feedback input" (col. 7, 
lines 52-54). In addition, Graham also teaches, "the affect is to automatically refine the 
patterns searched" (col. 7, lines 3-4). This teaches the system receiving user feedback 
input and refines the search. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to modify the teaching of Graham 
into Cullen in order to provide relevant feedback from the user and also allow the user 
to refine the search when the retrieval results are not relevant. 
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Regarding on claims 7 and 34, Cullen teaches presenting the first and second 
images to a user (col. 4, lines 14-27); however, Cullen does not explicitly teach 
receiving feedback from the user as to which of the first and second images are relevant 
to the query; and assigning a large weight to an association between the query keyword 
and the images deemed relevant by the user. On the other hand, Graham teaches, "the 
modification may occur automatically in the background or may involve explicit user 
feedback input" (col. 7, lines 52-54). In addition, Graham also teaches, "changing 
probability values, introducing a new connection between a subconcept and concept, or 
introducing a new keyword or key phrase" (col. 7, lines 62-65). This teaches using the 
user's feedback to change or assign the new weight for the keyword or images to create 
the new connection between the keyword and the image or the document: Therefore, it 
would have been obvious to one ordinary skill in the art at the time of the invention was 
made to modify the teaching of Graham into Cullen in order to allow the system to 
adjust the weighting process to determine which of the images are more relevant to the 
search query to enable the user to retrieve the better image for the next search. 

Regarding on claims 8 and 35, Cullen teaches grouping the low-level features of 
the images deemed by relevant by the user (col. 4, lines 18-27). 
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Regarding on claims 9 and 36, Cullen teaches presenting the first and second 
images to a user (col. 4, lines 14-27); However, Cullen does not explicitly teach 
receiving feedback from the user identifying an example image as less or irrelevant to 
the query for refinement of the search; and assign a small weight to an association 
between the query keyword and the example image. Graham teaches, "the modification 
may occur automatically in the background or may involve explicitly user feedback 
input" (col. 7, lines 52-54). This teaches the system is receiving user' feedback. In 
addition, Graham also teaches, "user 504 indicates the selected keywords or key 
phrases to be a little interest, the probability values connecting these keywords or key 
phases to the concept decrease" (col. 8, lines 10-13). This teaches the value of the 
keyword associated with the concept is decreased if the keyword is not descriptive. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to modify the teaching of Cullen into Graham because by 
deceasing the value of a keyword associated with concept or image would allow the 
user to search again with the same keyword without receiving the same irrelevant 
concept or image. 

Regarding on claims 10 and 37, Cullen teaches identifying additional images with 
low-level features similar to those of the example image (col. 4, lines 18-27). 
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5. Claims 12-15 and 39-41 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Barber (US. Patent No. 5,579,471 ). 

Regarding on claims 12, 15, 39 and 41 , Barber teaches a method comprising: 

Permitting entry of both keyword-based queries (keyword) and content-based 
queries (image query) (col. 3, lines 23-38); 

Finding images using both semantic-based image retrieval and low-level feature- 
based image retrieval (col. 5, lines 30-42); 

Presenting the images to a user so that the user can indicate whether the images 
are relevant (col. 5, lines 30-42); and 

Barber does not explicitly teach conducting semantic-based relevance feedback 
and low-level feature-based relevance feedback in an integrated fashion. However, 
because by utilizing the relevancy feedback would allow user to change the association 
between the keyword and the concept or image to allow the user to retrieve better 
result. Lang teaches, "user 5 provided with a multiple feedback queries along with the 
proposed informon. By a answering, user 5 creates a feedback profile that corresponds 
to feedback response 25. User feedback response 29 can be active feedback, passive 
feedback, or a combination" (col. 7, lines 47-52). This teaches the user give the 
feedback on the search results. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to combine the teaching of Lang 
and Barber because utilizing the feedback by the user would allow the system to learn 
and modify the images. 
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Regarding on claims 13 and 40, Barber teaches ranking the images (col. 12, 
lines 30-33). 

Regarding on claim 14, Barber teaches using images indicated as being relevant 
to find additional images (col. 14, lines 46-47). 
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6. Claims 16-19 and 38 are rejected under 35 U.S.C. 103(a) as being unpatentable 

over De Bonet (US. Patent No. 5,899,999). 

Regarding on claims 16, 19 and 38, De Bonet teaches a method comprising: 
associating keywords with images to form keyword-image links (col. 22, lines 53- 

57); 

Assigning weight to the keyword-image links (col. 8, lines 62-64); 

Presenting a result set of images obtained from an image retrieval search based 
on a query (col. 22, lines 59-62); 

De Bonet does not explicitly teaches receiving feedback from the user as to 
whether the images in the result set are relevant to the query; and modifying the weight 
according to the user feedback. However, De Bonet teaches, "for example if a desired 
image is to have less variant in color across the image than does the test image, then 
the user will ostensibly choose a relatively low weight for color distribution, and so forth 
for other weights... To retrieve an image, the system compares the vector for the test 
image, modified by the weight provided by the user, to the vector for each image in the 
database" (col. 4, lines 48-56). This teaches the system takes the user feedback and 
adjusts the weight to the image. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to modify Bonet's system to include 
the use of taking user feedback and modified the weight to the image in order to provide 
the better results for other user to search. 
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Regarding on claim 17, De Bonet teaches the modifying comprises increasing 
the weight of a keyword-image link for images deemed by the user as more relevant to 
the query (col. 4, lines 59-64). 

Regarding on claim 18, De Bonet teaches the modifying comprises decreasing 
the weight of a keyword-image link for images deemed by the user as less relevant to 
the query (col. 4, lines 59-64). 
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7. Claims 20 and 22-23 are rejected under 35 U.S.C. 103(a) as being unpatentable 
overGustman (US. Patent No. 6,092,080). 

Regarding on claims 20 and 23, Gustman teaches the a method comprising: 
Presenting a result set of images that are returned that are returned from an 
image retrieval search of query having at least one keyword (col. 9, lines 40-50); 

Gustman does not explicitly teach monitoring feedback from a user as to 
whether the image in the result set are relevant to the query; in an event that the user 
selects at least one image as being relevant to the query, associating the keyword in the 
query with the selected image to form a first keyword-image association and assigning a 
comparatively large weight to the first key-image association; and in an event that the 
user identifies an example image for refinement of the search, associating the keyword 
in the query with the example image to form a second keyword-image association and 
assigning a comparatively small weight to the second keyword-image association. 
However, Gustman teaches, "a quality assurance mechanism can be used to monitor 
the quality of the input data and provide feedback" (col. 17, lines 23-25). This teaches 
the feedback mechanism. In addition, Gustman teaches, "an instance phrase 706 can 
be related to a set of proposed keywords, a set of keywords, a set of images and/or 
video, a set of proposed person, and a set of persons, each set having zero or more 
members" (col. 17, lines 59-62). This teaches associating the keyword with image. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to modify Gustman's system to include the use of taking user 
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feedback and associating of keywords with the image in order to provide the system to 
improve the search results. 

Regarding on claim 22, Gustman teaches presenting the result set of images in a 
user interface, the user interface facilitating the user feedback by allowing the user to 
indicate which images are more relevance (col. 7, lines 51-65) and which images are 
less relevant (col. 17, lines 23-25). 

8. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Gustman (US. Patent No. 6,092,080) in view of Barber (US. Patent No. 5,579,471). 

Regarding on claim 21, Gustman does not explicitly teach conducting both 
content-based image retrieval and semantic-based image retrieval. However, Barber 
teaches, "a computer interface for constructing an image query used to access images 
in a database based on image content includes a first selection window for specifying 
image color and including a plurality of color thumbnail..., and a fourth selection window 
including a plurality of category thumbnails which may denote keywords, text, or 
conditions on alpha-numeric data associated with an image" (col. 3, lines 23-38). This 
teaches the inputted query with keywords and feature. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention was made to combine 
teaching of Gustman and Barber because utilizing the both keyword and feature would 
allow the search to be conducted more thoroughly. 
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9. Claims 24-25 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Nojima (US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6,504,571). 

Regarding on claim 24, Nojima teaches a method comprising: 

Computing, for each category, a representative feature vectors of a set of 
existing images with the category (col. 4, lines 30-41 ); 

Determining a set of representative keywords that are associated with the 
existing images in each category (col. 5, lines 50-57); 

Comparing, for each new image, the low-level feature vectors of the new image 
to the representative feature vectors of the existing images in each category to identify a 
closet matching category (col. 1 , lines 19-37); and 

Nojima does not explicitly teach labeling the new image with the set of 
representative keywords associated the closet matching category; however, 
Narayanaswami teaches, "the system also includes an image annotation module 220, 
operatively connected between the user interface/display 202 and the images from the 
image database 216, which allows a user to retrieve digital images with additional 
parameters or keywords such as names and descriptions of objects in the images" (col. 
8, lines 52-58). This teaches the annotation module annotates the keywords to the 
images. Therefore, it would have been obvious to one ordinary skill in the art at the 
time of the invention was made to combine the teaching Nojima and Narayanaswami 
because utilizing the annotation module to annotate the keyword to the images would 
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allow the images to be retrieved by other keywords that are more descriptive to the 
images. 

Regarding on claim 25, Narayanaswami teaches using feedback to selectively 
add and/or remove (annotating means adding) keywords from the new image (col. 8, 
lines 52-58). 

10. Claim 26 is rejected under 35 U.S.C. 103(a) as being unpatentable over Nojima 
(US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6,504,571) and futher in view of Kopec etal. (US. Patent No. 5,594,809). 

Regarding on claim 26, Both Nojima and Narayanaswami teach placing the 
labeled new images into a holding category (col. 8, lines 53-57). 

Both Nojima do not explicitly teach evaluating the labeled new images in the 
holding category to determine if any of the keywords associated with the labeled new 
image match the representative keywords from each category; and assigning the 
labeled new image to the category that best matches the keywords associated with the 
labeled with the labeled new image. 

Spec teaches evaluating the labeled new images in the holding category to 
determine if any of the keywords associated with the labeled new image match the 
representative keywords from each category (col. 7, lines 63-65); and assigning (input 
keyword image) the labeled new image to the category that best matches the keywords 
associated with the labeled with the labeled new image (col. 7, lines 64-65). 
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Therefore, it would have been obvious to one ordinary skill in the art at the time 
of the invention was made to employ the teaching of Kopec into Nojima and 
Naraynaswami in order to match the image with the keyword to allow the user search 
for these images with the same keyword. 

1 1 . Claim 29-30 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Nojima (US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6,504,571 ) and further in view of Stuckey et al. (US. Patent No. 5,721 ,938). 

Regarding on claim 29, both Nojima and Narayaswami do not explicitly teach the 
query handler comprises a natural language parser; however, Stuckey teaches, 
"present program provides a natural text parser, which may be used for all natural 
languages" (col. 3, lines 22-24). This teaches the query is inputted by the natural 
language. Therefore, it would have been obvious to one ordinary skill in the art at the 
time of the invention was made to modify the teaching of Stuckey into Nojima and 
Narayanaswami because utilizing the natural language parser would allow the user to 
input the natural language to search and to provide a user-friendly system. 

Regarding on claim 30, Stuckey teaches the query handler comprises: 

A parser to parse text-based queries (col. 3, lines 22-24); and 

A concept hierarchy to define various categories of images (col. 12, lines 22-25). 
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Conclusion 

1 2. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Baoquoc N. To whose telephone number is (703) 305- 
1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM - 4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kim Y. Vu can be reached at (703) 305-4393. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 
Commissioner of Patents and Trademarks 
Washington, D.C. 20231. 

The fax numbers for the organization where this application or proceeding is 
assigned are as follow: 

• (703)746-7238 [After Final Communication}] 

• (703) 746-7239 [Official Communication] 

• (703) 746-7240 [Non-Official Communication] 
Hand-delivered responses should be brought to: 

Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 



Fourth Floor (Receptionist). h£fiiM Corr, ELUS 

PRIMARY EXAMINER 





Baoquoc N. To 
April 16, 2002 
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